The present invention relates to computational processing. More particularly, the present invention provides systems and methods for distributing a dataset to a plurality of nodes, which in turn produce node specific outputs from the dataset. As just one example, the dataset can be a group of seismic data that is processed by a plurality of nodes to form a three-dimensional image of a geologic formation. Other data sets, however, can be used.
The seismic processing industry is involved in processing large amounts of seismic data using very complex algorithms. This is especially true for imaging algorithms that use the majority of the compute power in this industry. Such processing has historically involved the use of expensive supercomputers, or high end workstations. As an example, the amounts of data involved and the complexity of the algorithms often requires weeks of processing time to create an image of a geologic structure. To reduce processing costs, the seismic processing industry has been experimenting with the use of low cost computers as its main processing engines. To control the time required to complete a processing job, clusters of such computers have been proposed. As one example, a number of dual-processor personal computers serving as compute nodes each can be connected to a data server in a star configuration.
The advantage of such a cluster is its low cost, and its scalable, high compute capacity. However, the interconnect between the data server and the various nodes is often a bottleneck limiting the compute capacity of such a cluster. The problems associated with this bottleneck increase as the size of a cluster increases. As one example, a cluster may exceed one thousand nodes. One solution is to split a dataset and to provide multiple servers that each provide access to at least a portion of the dataset via an elaborate switch configuration. Such an approach requires relatively expensive switch hardware, and implementation of a complex data location monitoring scheme to assure data coherency and to identify which server supplies which portion of a given dataset. Another solution is to simply increase the bandwidth of a network switch coupling a data server to a surrounding cluster of computer nodes. Such an approach significantly increases the cost of the switch, and ultimately results in the same bottleneck where the number of nodes is increased.
Additionally, imaging software used by the seismic processing industry typically utilizes some middleware, like Message Passing Interface (MPI), Parallel Virtual Machine (PVM), or Beowulf. These software tools require that each computer node in a cluster run exactly the same imaging executable. This can mean that the compute nodes must all be running the same operating system, and the same version of the middleware. A seismic processing company may thus be forced to purchase specific computers and software in an effort to create a homogeneous cluster. This inhibits the company's ability to upgrade and/or to select computers offering the best value.
Yet further, a number of seismic imaging applications utilize the Network File System (NFS) to deliver the data from the data server to the cluster nodes. NFS has the advantage to the software developer in that it makes file systems on the server appear to be local to the compute nodes. This reduces the programming effort needed to message pass the data within the application. However, NFS has two large disadvantages. The first is that it uses the network very inefficiently and puts a large load on the data server. This reduces the ability of the data server to deliver data to the cluster nodes. The second problem is that NFS is unstable. NFS links from one computer to the next and can “hang”, or fail to respond. This can cause jobs running on the cluster to fail. The probability of this occurring increases rapidly as the number of compute nodes increases. As imaging jobs can run for several weeks before completing, any failure can result in the loss of considerable time and effort.
Thus, there exists a need in the art to provide advanced systems and methods for computational processing. As will be appreciated from the following disclosure, the systems and methods according to the present invention address these, and a number of other problems related to processing.